Method and apparatus for determining health status

ABSTRACT

A system and method for monitoring the state of an individual. The method includes providing a stimulus to the individual, measuring a response to the provided stimulus, comparing the measured response to an expected response, and diagnosing one or more aspects of disease in accordance with the result of the comparison between the measured response and the expected response. The stimulus may be a predetermined test sequence, such as a visually displayed predetermined sequence of images, or may include observation of the physical response of the individual while performing one or more predetermined activities. Stored images or video of the individual responding to one or more test sequences may be stored in a lossy or lossless state, and thus security and de-identification may be provided to stored data. This stored data may also be de-identified in a manner to allow for the answering of the greatest number of future questions.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/682,366, filed Aug. 21, 2017, which claims the benefit of thefollowing U.S. Provisional Patent Applications, the entire contentsthereof being incorporated herein by reference:

U.S. Provisional Patent Application Ser. No. 62/377,818, filed Aug. 22,2016, to Hanina, titled “Method and Apparatus for Determining HealthStatus”;

U.S. Provisional Patent Application Ser. No. 62/419,763, filed Nov. 9,2016, to Hanina et al., titled “Method and Apparatus for StoringInformation”;

U.S. Provisional Patent Application Ser. No. 62/505,627, filed May 12,2017, to Hanina at al., titled “Method and Apparatus for Storing andProcessing Information”; and

U.S. Provisional Patent Application Ser. No. 62/531,703, filed Jul. 12,2017, to Hanina et al., titled “Method and Apparatus for VisualDiagnostics.”

This application also incorporates by reference the entire contents ofthe material presented in U.S. patent application Ser. No. 13/189,518,filed Jul. 24, 2011, to Hanina et al., titled “Method and Apparatus forMonitoring Medication Adherence” and published as U.S. Patent App. Pub.No. 2012/0316897.

FIELD

The subject matter of the present disclosure relates generally tomonitoring patient health status and to the diagnosis and monitoring ofdisease employing visual images and image analysis, and moreparticularly to visually monitoring health status by visually orotherwise recognizing one or more characteristics of the patient, and tothe use of particular visual analysis tools and models in both activeand passive monitoring techniques to diagnose and monitor disease andsymptoms of disease as related to particular disease states and patientpopulations.

The subject matter of the present disclosure additionally relatesgenerally to storing personally identifiable visual information, andmore particularly to storing such video information in a manner thatallows for analysis of activity, clinical parameters, diagnosis ormonitoring of disease, or other desired features, in addition to anydesired future analysis of such data, while maintaining the security andprivacy of the data. The subject matter of the present disclosure alsorelates to the use of such stored secured information in the diagnosisand treatment of disease, as well as more generally determining statusof an individual based upon the stored data, as noted above.

BACKGROUND

Monitoring patient health status, as well as the diagnosis andmonitoring of disease, whether, e.g., during a clinical drug trial,during disease management by a physician, or in home care scenarios, mayentail confirming that a patient has administered required medication.

SUMMARY

In U.S. Pat. Nos. 8,731,856, 8,731,961, 8,666,781, 9,454,645, and9,183,601, the contents of these five patents being incorporated hereinby reference in their entirety, the inventors of the subject matter ofthe present disclosure have proposed a system, method and apparatus thatallow for complete control and verification of adherence to a prescribedmedication protocol or machine or apparatus use in a clinical trialsetting, whether in a healthcare provider's care, or whenself-administered in a homecare situation by a patient.

In U.S. Pat. No. 9,256,776 (the '776 patent), the content of this patentalso being incorporated herein by reference in its entirety, theinventors of the subject matter of the present disclosure describe asystem for de-identification of one or more images of an individual. Thetechniques for determination of one or more portions of an image to bede-identified may be applied to the subject matter of the presentdisclosure, thus reducing required computing load. As will be describedbelow, application of the feature extraction/keypoint method and systemmay be applied not necessarily to complete images, but additionally toone or more portions of images as identified in accordance with the '776patent.

These patents and other patents attributable to the inventors of thesubject matter of the present disclosure present the only medicationmanagement system that may determine whether a user is actuallyfollowing a protocol, provide additional assistance to a user, startingwith instructions preferably including one or more interactive andreal-time audio, visual, textual or the like prompts based upon one ormore actions detected of the user, and moving up to contact from amedication administrator if it is determined that the user would needsuch assistance in any medical adherence situation, including clinicaltrial settings, home care settings, healthcare administration locations,such as nursing homes, clinics, hospitals and the like, and in clinicaltrial settings. They also present the only system designed tocontextually de-identify images of the face of an individual allowingfor review of these images while maintaining the security of theseimages.

The subject matter disclosed herein builds on these initial inventionsand provides one or more features that may be employed in accordancewith these systems to use further visual information collected by avideo camera or other sensor to determine additional characteristicsassociated with medication administration or otherwise the health of thepatient administering the medication, or any other individual.

The subject matter disclosed herein builds on these initial systems,devices and techniques, and additionally provides one or more mechanismsfor collecting and storing visual data related to the users of thesystem, and in a particular embodiment, video of the users administeringmedication. In this context, such video may provide a view of the faceof a particular user, thus allowing their identity to be determined fromthese video images. Such determinations may be made on the local deviceof a user, or at a remote processing location, such as a cloud computinglocation, or dedicated remote server location. While this informationmay be stored in a secure manner, through encryption and the like, theinventors of the subject matter disclosed herein have determined that itwould be beneficial to implement a system that balanced the need tosecure the video data while allowing for future analysis of the data inone or more manners currently not contemplated.

Therefore, in accordance with one or more embodiments of the presentdisclosure, a system and method are provided in which a video sequenceis preferably analyzed to determine a number of features that may berepresentative of one or more images in a video sequence, such askeypoints or landmarks of the face of a user indicative of the identity,a current health or other state of the user, or other visual features inone or more additional images. Once such features are determined, asubsequent determination may be preferably made to determine a subset ofthese features that may be maintained which allow for analysis of thevideo sequence, while being “lossy” enough to de-identify the images byprecluding reverse processing of the features to allow identification ofthe user. Images for use in accordance with the invention may becaptured using a camera capture device, such as that in a dedicatedcamera, a mobile device such as a smartphone, or the like. Analysisprocessing may employ any methods, such as computer vision analysis,neural networks, deep (reinforcement) learning, machine learning, or thelike. Processing may be provided by a processor embedded within such acamera, mobile device, or dedicated computing system. Data transmissionpreferably takes place over a cellular, Wi-Fi enabled or other wirelessor wired communication system.

Additionally, information retrieved from one or more users of thesystem, whether provided in identifiable or de-identified format, may beused in order to make determinations of diagnosis or monitoring ofdisease, or other detailed status of an individual, such asdetermination of pulse rate or other bodily states. Thus, any imagesobtained of such users may be relied upon in order to determine one ormore statuses of a user, whether the images are identifiable, orde-identified. The ability to perform such analysis from a de-identifiedimage ensures proper data protection while allowing for extensivepost-hoc analysis of these de-identified images. These images to beanalyzed may further be analyzed in other than collected chronologicalorder, this allowing for the analysis of individual frames, or frameshaving similar characteristics, together regardless of when these imageswere acquired.

The subject matter of the present disclosure further provides one ormore mechanisms for collecting and storing visual data related to theusers of a system, and in a particular embodiment, video of users orpatients performing one or more predetermined activities (activemonitoring), or performing their daily routine activities (passivemonitoring). Active monitoring may comprise presentation of a particularvisual sequence on a display, and monitoring of a user's response tothis visual sequence, such as by tracking or otherwise analyzing theeyes, gaze direction or other facial characteristic of the user. Passivemonitoring may comprise observing the user when performing a sequence asinstructed by a device, but for the purpose of another activity, ratherthan the visual sequence displayed as noted above, or by monitoringdaily routines that are known to the system in order to measure aresponse. In this context, such video may provide a view of the face ofa particular user, thus allowing for one or more desired characteristicsto be viewed and tracked over time. Such characteristic review may bemade on the local device of a user, or at a remote processing location,such as a cloud computing location, or dedicated remote server location.This information is preferably stored in a secure manner, throughencryption and the like.

Therefore, in accordance with one or more embodiments of the presentdisclosure, a system and method are provided in which a video sequenceof a user performing one or more predetermined activity sequences, orperforming routine activities, is analyzed to determine a number offeatures that may be representative of one or more diagnosticattributes, such as eye movement, affect, heartrate, skin tone and thelike. Once such features are recognized and tracked, a subsequentdetermination may preferably be made to determine a subset orcombination of these features that are indicative of diagnosis ormonitoring of disease, and may be analyzed over time to determinechanges in a particular disease progression.

The subject matter of the present disclosure provides one or morefeatures that may be employed in accordance with these systems to usefurther visual information collected by a video camera or other sensorto determine additional characteristics associated with the health of apatient, or any other individual.

Images for use in accordance with the subject matter of the presentdisclosure may be captured using a camera capture device, such as thatin a dedicated camera, a mobile device such as a smartphone, or thelike. Analysis processing may employ any methods, such as computervision analysis, neural networks, deep learning, machine learning, orthe like. Processing may be provided by a processor embedded within sucha camera, mobile device, or dedicated computing system, either local orremote, such as in a cloud-based system. Data transmission preferablytakes place over a cellular, Wi-Fi enabled or other wireless or wiredcommunication system.

Additionally, information retrieved from one or more users of the systemmay be used in order to make determinations of diagnosis of disease, orother detailed status of an individual, such as determination of pulserate, eye movement, or other bodily states. Thus, any images obtained ofsuch users may be relied upon in order to determine one or more statusesof a user.

Implementations of the subject matter of the present disclosure may haveadvantages relative to existing systems and techniques. For example,U.S. Pat. No. 7,359,214 includes a device that provides instruction to apatient regarding medications to take. This system, however, provides nomechanism for actually confirming that a patient is in fact properlyadministering required medication, including, e.g., placing a medicationpill into their mouth, or injecting or inhaling medication following apredetermined series of steps, as required in a clinical drug trial, asprescribed by a prescribing physician in the case where adherence to aparticular regimen may prove to be critical to efficacy of theprescription regimen, in various public health scenarios, in situationswhere failure to keep up a prescription regimen can potentially harm apopulation as a whole, such as the generation of antibiotic-resistantbacteria strains, in various disease management scenarios, or in homecare situations where maintaining proper control of administeringhealthcare professionals is critical. U.S. patent application Ser. No.11/839,723 (now U.S. Pat. No. 8,538,775), filed Aug. 16, 2007, titledMobile Wireless Medication Management System provides a medicationmanagement system employing mobile devices and an imaging technology sothat a user is able to show a pill to be taken to the system, and thesystem can then identify the medication. Similarly, however, there is infact no particular manner in which to ensure actual adherence, includingingestion, inhalation, injection of the medication, or the relationshipof adherence to the efficacy or safety of the drug over time.

The inventors of the subject matter disclosed herein have determinedthat diagnosis and monitoring of disease traditionally requiressubjective determination of disease state by a healthcare professional.Application of known disease parameters to a currently observed set ofdisease states from a patient results in a diagnosis of disease.Continued monitoring of these disease states allows for monitoring ofdisease, and determinations of progression thereof, over time. Medicaldiagnosis and monitoring systems typically rely on such subjectivedeterminations, or upon measurements made in a controlled environment,such as blood draws in a clinic, x-rays and the like.

The inventors of the subject matter disclosed herein have furtherdetermined, however, that prior art systems fail to describe the use ofadvanced visual analysis to properly diagnose and monitor disease,collecting information from across populations, and determining criticalcharacteristics indicative of such diagnoses. These systems similarlyfail to take into account accurate determinations of medicationadherence in order to further diagnose and monitor disease.

U.S. Pat. No. 8,408,706 entitled “3D Gaze Tracker” describes a systemfor determining a gaze of a person, comprising a 3D camera that acquiresa range image of a person, a picture camera that acquires a picture ofthe person, and a controller that processes images acquired by thecameras to determine a gaze direction and origin for the gaze vector ofan eye of the person. This complex system is one that is difficult orimpossible to implement on a simple mobile device, and requires complexprocessing of an image being viewed by the person in order to determinethe input stimulus being provided to the person.

US Patent Application Serial No. 2013/0321772 entitled “MedicalDiagnostic Gaze Tracker” describes a system for detecting a gazebehavior of a user in response to a stimulus, where the stimulus is anon-diagnostic stimulus, determining a type of the stimulus, andgenerating an indication of the gaze behavior and the type of thestimulus. Thus, while the system is not an active system (i.e., does notprovide a predetermined stimulus to the subject) it does require acomplex system for imaging the field of view being viewed by a person,and determining a type of input stimulus being observed by the person.

International Patent Application Serial No. WO/2001/074236 describes asystem for diagnosing Attention Deficit Hyperactivity Disorder (ADHD).The application in turn relies upon a complex eye tracker systemdescribed in U.S. Pat. No. 4,852,988, and comprises a complex systemincluding a helmet to be worn by the person being tested. While thisapplication describes a number of characteristics to be reviewed indiagnosing ADHD, the system suffers from complexity, and the inabilityto monitor eye movement when viewing a passive, non-predetermined set ofvisual images, as the system is primarily designed to track the eyes ofthe person while performing a physical task.

Additionally, existing systems fail to further incorporate themonitoring of any further patient characteristics to aid in determiningproper medication adherence, and to further determine a status of thepatient.

The inventors of the subject matter of the present disclosure havefurther determined that these and other existing systems fail toconsider that sensitive or other individually identifiable informationmay be captured and stored by these systems. These existing systems failto consider confidentiality of any acquired information, and the abilityto store this information in a secure, confidential manner while stillallowing for future analysis of any such acquired and stored data.

The inventors of the subject matter of the present disclosure havefurther determined that prior art systems fail to describe the use ofsuch stored data for the purpose of diagnosing or monitoring disease orcurrent status of an individual.

Still other objects and advantages of the invention will in part beobvious and will in part be apparent from the specification anddrawings.

The subject matter of the present disclosure accordingly comprisesseveral steps and the relation of one or more of such steps with respectto each of the others, and the apparatus embodying features ofconstruction, combinations of elements and arrangement of parts that areconfigured to affect such steps, all as exemplified in the followingdetailed disclosure, and the scope of the invention will be indicated inthe claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Reference is made to the following description and accompanyingdrawings, in which:

FIG. 1 is a graph depicting information and a tradeoff between levels ofprivacy and the ability to perform tasks on the information inaccordance with an exemplary embodiment of the invention;

FIG. 2 is a graph depicting a relationship between computational costand relevant information quality for various combinations of storedinformation in accordance with an exemplary embodiment of the invention;

FIG. 3 depicts various levels of keypoint de-identification inaccordance with an exemplary embodiment of the invention;

FIG. 4 depicts a relationship between various sensors, measurements anddisease determinations in accordance with an exemplary embodiment of theinvention;

FIG. 5 depicts an additional relationship between the items depicted inFIG. 4, and in particular physical implementation of one or more of therelationships thereof;

FIG. 6 depicts an exemplary embodiment showing the overlaying of a knownphysical structure on the face of an individual, thereby improving theability to recognize one or more physical changes in the physicalstructure;

FIG. 7 depicts a structure for feature extraction on multiple sensedlayers in accordance with an exemplary embodiment of the invention;

FIG. 8 is an exemplary embodiment of the invention depicting the layersand features that may preferably be extracted when monitoring medicationadherence of a user;

FIG. 9 depicts selective keypoint de-identification in accordance withan exemplary embodiment of the invention;

FIG. 10 depicts a de-identification model, including a method forgenerating de-identified images which are compatible with existingmodules, in accordance with an exemplary embodiment of the invention;

FIG. 11 depicts a method for training the de-identification model ofFIG. 10 to optimize the parameters thereof, in accordance with anexemplary embodiment of the invention;

FIG. 12 depicts a joint optimization de-identification model constructedin accordance with an exemplary embodiment of the invention;

FIG. 13 depicts a joint optimization method for training the jointoptimization de-identification model of FIG. 12 to optimize theparameters thereof in accordance with an exemplary embodiment of theinvention;

FIG. 14 depicts a possible approximation for joint optimization of thetwo models (FIGS. 10 and 12), and operates in a block-wise optimizationof the parameters in accordance with an exemplary embodiment of theinvention;

FIG. 15 is a block diagram depicting the details of an exemplaryhardware configuration for implementing the subject matter of thepresent disclosure;

FIG. 16 is a block diagram depicting additional details of an exemplaryhardware configuration for implementing the subject matter of thepresent disclosure; and

FIG. 17 is a flowchart diagram depicting additional details of anexemplary implementation of monitoring to determine disease progression.

DETAILED DESCRIPTION

In accordance with an embodiment of the subject matter of the presentdisclosure, a visual motion capture device, camera or the like is usedto capture motion information related to the administration of pill orfilm based oral medications, or injectable, inhaler-based, othernon-pill based medication, or any other form of patient administrationtask that may be performed, may be utilized in accordance with one ormore of the methods, devices, and systems noted in the above-referencedapplications.

Information Capture System

Referring first to FIG. 15, a remote information capture apparatus 1000is shown. Such apparatus is configured to allow for the capture andprocessing of information in order to implement the system and method inaccordance with the subject matter of the present disclosure. Suchinformation capture apparatus 1000 is preferably placed in communicationwith a remote data and computing location 3000 via a communicationsystem 2000, preferably the Internet or other communication system. Viacommunication system 2000, information captured by apparatus 1000 may betransmitted to remote data and computing location 3000, and analysisinformation or other instructions may be provided from remote data andcomputing location 3000 to apparatus 1000. It is further contemplatedthat a plurality of such information capture apparatuses 1000 may becoordinated to monitor a larger space than a space that can be coveredby a single such apparatus. Thus, the apparatuses can be made aware ofthe presence of the other apparatuses, and may operate by transmittingall information to one of the apparatuses 1000, or these apparatuses mayeach independently communicate with remote data and computing location,which is configured to piece together the various information receivedfrom the plurality of devices 1000.

Referring next to FIG. 16, a more detailed view of a preferredembodiment of remote information capture apparatus 1000 and remote dataand computing location 3000. As is shown in FIG. 16, apparatus 1000comprises an information capture device 1110 for capturing video andaudio data as desired. A motion detector 1115 or other appropriatetrigger device may be provided associated with capture device 1110 toallow for the initiation and completion of data capture. Informationcapture device 1110 may comprise a visual data capture device, such as avisual camera, or may be provided with an infrared, night vision, orother appropriate information capture device. A storage location 1120 isfurther provided for storing captured information, and a processor 1130is provided and configured to control such capture and storage, as wellas other functions associated with the operation of remote informationcapture apparatus 1000. An analysis module 1135 is provided inaccordance with processor 1130 and configured to perform a portion ofanalysis of any captured information at the remote information captureapparatus 1000. Apparatus 1000 is further provided with a display 1140,and a data transmission and receipt system 1150 and 1160 for displayinginformation, and for communicating with remote data and computinglocation 3000. Remote data and computing location 3000 comprises systemmanagement functions 3030, and a transmission and reception system 3050and 3060 for communicating with apparatus 1000. Transmission andreception system 3050 and 3060 may further comprise various GPS modulesso that a location of the device can be determined at any time, and mayfurther allow for a message to be sent to one or more individualapparatuses, broadcast to all apparatuses in a particular trial, orbeing used for administration of a particular prescription regimen, ofbroadcast to all available apparatuses.

In accordance with an embodiment of the present disclosure, apparatus1000 is preferably configured to be part of a system that improvesadherence to a medical protocol, and to analysis of collected visualdata of user medication adherence to determine health status of users.Users of apparatus 1000 in accordance with this system giveadministrators a tangible and concrete manner in which to reviewactivities and collected information. Apparatus 1000 of the invention isconfigured to receive instructions for patients from remote data andcomputing location 3000 and provide these instructions to patients. Suchinstructions may comprise written, audio or audio instructions forguiding a user to perform one or more activities, such as performing asequence of actions to test a particular action of the user, or whethera user is adhering to a prescribed medication protocol.

The system, in accordance with an embodiment of the present disclosure,is also applicable to monitoring of patient activities when beingrequested to perform particular actions, or when performing routineactions not specifically requested. Therefore, in accordance with anembodiment of the present disclosure, a method and apparatus may beprovided for analyzing captured patient motion data, preferably in nearreal time to provide feedback to the user, to determine a number oftimes a participant performs some action that is, for example,considered suspicious, or to determine one or more elements ofdiagnosing or monitoring disease.

In accordance with a further embodiment of the present disclosure, thevisual capture device 1110 may be used to capture visual informationrelated to one or more subjects. Any standard camera or image capturedevice may be employed, including but not limited to a camera on amobile phone, tablet, other computing device, standalone camera, or anyother image acquisition apparatus that is able to record one or more(video) images of a subject. In a preferred embodiment of the presentdisclosure, the subject may comprise a face of a human, but may compriseany other desirable subject, including one or more other body parts ofthe user, or other object. Analysis of these recorded images may beperformed currently, or the images may be stored for future analysis.Storage of such images may be performed local to the image captureapparatus, at a remote location, such as a dedicated storage location,or a cloud based storage system. Storage of these images, however, maypresent a security problem, and the identity of the subjects or otherconfidential information in the captured visual information may bediscernable from these images. De-identification of this storedinformation will be described below.

Additionally, visual representations of a user can be further used todetermine a status of the user. Visual determination of one or moreparameters, such as motion, eye motion, skin tone, emotions, heart rate,blood pressure, body mass, gps location, proximity, or othermeasurements (such as non-visual measurements) that may be provided inaccordance with one or more incorporated or coupled sensor, may bemeasured visually or otherwise, at once or over time, to determinechanges in one more of such parameters in order to identify changes inthe health of the user. In accordance with an embodiment of the presentdisclosure, by way of example, display 1140 preferably displays one ormore bits of information to a user. Such information may preferablycomprise a specific video sequence designed to test the reaction of theuser, or may comprise interactive or other instructions to the user toperform a predetermined activity. Information capture apparatuspreferably captures information monitoring the user upon viewing of thedisplayed information, and performing one or more activities in responsethereto. Other devices for capturing information in response topresented visual or other stimuli may include diverse sensors, such aglucose meters, blood pressure cuffs, radar systems, visual capturedevices, thermometers, accelerometers (measuring the shake of the handof a user, for example), or the like. One or more of such measureparameters may be used to identify particular characteristics of one ormore disease states. In such a manner, while monitoring adherence orother activities, or when performing actions in response to a presentedtest script, such parameters may be measured, and reported back to oneor more healthcare professionals, one or more care providers, otherindividuals, or may be collected in order to analyze automatically,perhaps over time, to diagnose and monitor disease. Thus, theseparameters may be measured over time without reference to adherence,allowing for diagnosis of disease, measurement of progression of diseaseonce diagnosed, or measurement of various health indicators to gauge theoverall health of an individual.

Furthermore, a database or other repository of such measurements may becollected over time and over users at remote data and computing location3000. Such database may be characterized by disease state or otherdemographic characteristics. In this manner future measurements of oneor more users may be compared against such a database, and allow fordiagnosis of one or more diseases, or changes in these characteristicswhen monitoring these diseases. Furthermore, expected progression ofsuch parameters over time may be determined for a population as a whole,or for various portions of a population, defined by demographics,disease state or the like. So, by way of example, it may be possible todetermine expected progression of one or more visual characteristics,such as weight gain, of a female aged 40-60 suffering from diabetes, orto determine expected changes in response to a visual presentation of ascript to be followed by a user. Of course, other sub-populations mayalso be determined from such a database.

In an additional embodiment of the present disclosure, the measuredcharacteristics may comprise one or more de-identified video assets thatmay be determined from capture of images of a user from connected,integrated or other visual capture apparatus 1110, such as a still orvideo camera. De-identification of one or more of such captured imagesallows for further analysis of these images, generating a more robustdatabase, while protecting the identity of any individual originallypictured in the images. Thus, through the generation of derivativeimages, portions of images, characteristics or the like, information maybe stored in a manner that allows for future analysis while protectingthe identity of the users. One or more mechanisms for generating suchde-identified images will be described below.

In yet a further embodiment of the present disclosure, the determinationof whether a particular user has modified disease characteristics may bedetermined in accordance with one or more unsupervised learning systems,such as a neural network or the like. In such a manner, the database ofcollected images may be employed to train such a system, identifying oneor more characteristics from the training images that may be used toidentify similar characteristics in future images. Furthermore, the useof such unsupervised learning techniques preferably provides anadditional layer of de-identification of images, allowing for thefurther collection of images of users, and the subsequent maintenance ofthese images in a de-identified state through the generation ofderivative images, image portions, extracted characteristics and thelike that may be stored for comparison to future images processed in asimilar manner. Furthermore, data may be further encoded in theseimages, such as one or more codes associated with a camera device, orother identifying information. Additional information collected from oneor more external sensors, such as accelerometers, void recorders,associated with the camera device, or one or more external medicaldevices, such as glucose meters, heartrate meters, or other measurementdevices, or any of the sensors noted in FIG. 4 (as will be describedbelow) may be further included in the unsupervised learning system toadditionally encode or categorize images. This collected information maybe used to calibrate the system during a learning phase, and may besubsequently removed during an operation phase. Combination of one ormore of these readings with visual information may further allow for thedetermination of additional changes in status of a patient or user.

By way of example, pulse oxymiters, heartrate monitors and the like maybe employed with collected video information to allow for more precisedeterminations. Additionally, micro movements associated with movementof a mobile device or the like may be also be employed. Micro eyemovements, gaze tracking, analysis of expression, or any other microgestures, micro movements, or other recognizable conditions and the likemay further be employed. These additional measured features may befurther employed to identify changes in characteristics along a numberof alternative dimensions in such an unsupervised or supervised learningsystem, ultimately diagnosing or monitoring disease. Analysis of theaccumulated information may allow for identification of one or morecommon characteristics among or between various disease states,demographic states, or other common identifying characteristic.

Longitudinal analysis of such data and changes in visual and othercharacteristics over time may be further correlated to negative healthoutcomes such as hospitalization events or death, and may give rise torelationships that can then act as the basis to trigger interventions inadvance of a negative health outcome occurring. Through such monitoring,early warning signs may be extracted from visual images of users in amanner not previously possible. Thus, any number of visual analysistechniques may be employed to generate a video asset base by therapeuticarea over time, thus allowing for the user of such assets to evaluatethe health of users in the future including similar characteristics, andresiding in similar therapeutic areas.

In accordance with one or more embodiments of the present disclosure, itis anticipated that the use of one or more sections of theelectromagnetic spectrum will allow for an in-depth analysis of facialor other visible user features. For example, as will be described below,rather than simply noting external facial features, the techniques andsystems disclosed herein allow for the determination of the location ofvarious blood vessels under the skin of a user in the field of view of acamera. Over time, differences determined in the various images provideinformation about the performance of the user, and may further indicatechanges in disease, physical ability, or the like. Such changes, forexample, may be more visible under near-infrared light, or otherwavelength of energy, thus resulting in additional information beingextracted based upon the use of multiple types of light, energy, orother data extraction mechanisms.

As will similarly be described below, by overlaying multiple layers ofinformation, irrespective of the amount of information provided in eachof those layers, a more concrete picture of the status of an individualmay be provided. Information collected from lower power devices mayinclude lower resolution information, such as images with fewer pixels,or may include different mechanisms for data collection. As moreinformation is acquired, either through different collection mechanisms,or as the power of collection improves over time, correlations betweencollected data and negative outcomes (or positive outcomes) in one ormore medical therapeutic areas may be provided. The system may thereforebe employed as a diagnostic tool for predicting disease or otherdiagnosis based upon processing of visual images of a user. By allowingfor the use of such varied layers, the system is robust to methods ofcollection, timing of collection, and storage formats of theinformation, thus “future proofing” the system to rely on older data(perhaps at lower resolutions), and newer collected data, having ahigher resolution, or a newer or new data collection technique.

The inventive system may therefore learn various correlations betweenone or more observed features, and health status, health outcomes,disease progression, symptom progression, or one or more changes inoverall health. By analyzing and correlating these changes in featuresand ultimate health status, the system provides a mechanism fordetermining yet unknown relationships between measurable quantities andthe health of an individual. Once established, these relationships canthen be used to predict future medical situations. By way of example,one or more sub-segments of the population may be targeted forobservation. If such population is a post-stroke population, it is knownthat rapid weight gain may be a symptom of failure to take propermedication, or may predict a more urgent medical situation. Inaccordance with an embodiment of the present disclosure, daily images ofan individual may allow for a determination of such rapid weight gainwithout the use of a body-weight scale. In such a situation, ahealthcare provider may be immediately notified to follow up with theindividual. While visual-spectrum features may be used to determineweight gain, determinations of changes in pulse, blood pressure or othermeasurements may rely on the above-mentioned other areas of theelectromagnetic spectrum, audio pulses, or any other type of desirablesensor, whether alone or in concert with a visual analysis.

In accordance with alternative embodiments of the present disclosure,accumulated images of one or more users, associated sensor datainformation, visually extracted information, and one or more additionalinputs may be incorporated into a comprehensive database. Analysis ofthe accumulated information may allow for identification of one or morecommon characteristics among or between various disease states,demographic states, or other common identifying characteristic. Thus, inaccordance with one or more embodiments of the present disclosure,collected information may be stored in a standard or de-identifiedformat, and used to determine not only current conditions and actions,but also to diagnose disease, changes in health conditions, or tosegregate individuals from populations to identify one or more potentialmedical or other unique conditions. The techniques and systems disclosedherein provide the ability to perform these diagnoses from de-identifiedinformation, thus allowing for the protection of personal informationwhile retaining sufficient information to allow for subsequentdiagnosis.

As noted above, this accumulated information may be employed to trainone or more learning systems, thus further extracting commoncharacteristics or elements to be extracted. These extractedcharacteristics comprise one or more derivative elements of theaccumulated images and other data, and allow for the storage of thesederivative elements in a de-identified manner, without the need to storeand maintain the images including identifying information. In such amanner, large databases of patient information may be processed, andcharacteristics extracted therefrom may be used to identify furthercommon characteristics from later acquired images related to medicationadherence, or other disease state characterization. The system willtherefore be adaptable to diagnose disease, or other changes in patientstatus simply by taking images of a new user, either in a single shot,or noting changes in any measured characteristics over time.

Longitudinal analysis of such data and changes in visual and othercharacteristics over time may be further correlated to negative healthoutcomes such as hospitalization events or death, and may give rise torelationships that can then act as the basis to trigger interventions inadvance of a negative health outcome occurring. Through such monitoring,early warning signs may be extracted from visual images of users in amanner not previously possible. Thus, any number of visual analysistechniques may be employed to generate a de-identified video asset baseby therapeutic area over time, thus allowing for the user of such assetsto evaluate the health or users in the future including similarcharacteristics, and residing in similar therapeutic areas.

The system may process information at remote system 3000 housing adatabase of collected information. New images acquired by an imageacquisition camera 1110 in local mobile device 1000 may be transmittedto the remote location 3000, one or more of the above-notedidentification or processing techniques may be applied, and then theresults of such analysis may be provided as feedback to a user or otherhealthcare provider to provide advanced notice for possible adverseevents, changes in health, and predicted outcomes, such as futurehospitalizations, worsening of illnesses, particular symptoms or thelike. Alternatively, processing may be performed in advance of imageacquisition, and then a system for identifying characteristics of afuture image may be provided to the local device to further processacquired images. Such a system may be reduced in processing requirementsto allow for appropriate processing on the local mobile device. Byproviding either of these systems in accordance with a medicationmonitoring process, when monitoring medication administration, otherhealth issues of the user may be diagnosed. Such diagnosis may identifychanges in characteristics of a particular disease state, or mayidentify new diseases or one or more characteristics of any of suchdiseases.

De-Identification of Information

Referring next to FIG. 1, a graph depicting a relationship between alevel of privacy maintained vs. the ability to perform various desirabletasks is shown. In accordance with an embodiment of the presentdisclosure, captured information may include one or more images of theface of a user but may comprise an image of any other object. As isshown in FIG. 1, graph 110 a first horizontal axis 115 corresponds to alevel of privacy provided by a particular solution. As one moves furtherfrom the origin along horizontal axis 115, an increased level of privacyis provided. A second vertical axis 120 depicts an achievable level ofperformance on various tasks. Movement further from the origin alongvertical axis 120 represents an increase in the level of performanceattainable. The vertical axis is further partitioned in two. The bottom(closed) segment corresponds to performance on a fixed set of “known”tasks. The top (half open) segment corresponds to performance on othertasks which we might be interested in down the line but which are not inthe “known” set yet.

As is further shown in FIG. 1, four different scenarios for data storageare provided, with a corresponding performance available depicted. Afirst option is to maintain all collected information in a losslessrepresentation. This option is noted by star 125, and is positioned ongraph 110 high on vertical axis 120, representing a high flexibility inprocessing of data (as all information is available), but scores low onhorizontal axis 115 as there is no security provided by processing ofthis information. This option allows one to attain the best achievableperformance on any task known or unknown since none of the informationis discarded. For the same reason it provides no inherent privacy.

A second option denoted by lossy triangle 130 allows for the computationof some generic level data layers that have identifying informationremoved, and also to use the collected original data to answer someknown questions before removal of the identifying information. As can beseen in FIG. 1, this option 130 provides relatively high performance onfuture tasks, and also performs better on the security front. Providedone make a good choice of layers (features/descriptors) this solutionallows one to compute answers to the known tasks (possibly limiting theselection of algorithms). If a comprehensive set of sufficiently genericfeatures are selected to be computed, it can also be expected that thisconfiguration is also able to perform well on a wide variety of unknowntasks, those which rely on information which is present in the genericlayers. In terms of privacy the layers can be designed to mask privateinformation. The more layers collected and the more generic they are,the more flexibility in answering future queries, but also the greaterthe possibility that some identifiable information may be gleaned fromthe stored information.

A third option 135 goes a bit further and simply computes answers toknown questions and then discards all of the initially-acquiredinformation. While this provides a further improvement along thesecurity axis, the ability to answer any future questions that mightarise is minimal, as there is no further representation of theinitially-acquired data to be processed. Computing answers to all knowntasks provides more privacy since the data stored is restricted to aminimal set of information. On the other hand, it is unlikely that allqueries will be determined up front, and therefore the ability toaddress many unknown future tasks is low.

Finally, option 140 simply discards all known data after any currentprocessing. While this situation provides the highest level of security,there is no ability to use the information in the future.

The inventors of the present disclosure have determined that option 130provides the best tradeoff between future processing and security. Thesubject matter of the present disclosure therefore presents a number ofsolutions in order to achieve this option along with a system allowingfor selectability of the security/future processing tradeoff.

Referring next to FIG. 2, in one embodiment of the present disclosure,the user, other individual, or automated process may make adetermination as to which level of security will be employed, and if aversion such as 130 is selected from FIG. 1, a determination of theamount of information and generic data that is to be stored must bemade. As noted with respect to FIG. 1, in addition to addressing theability to currently process information based upon available processingpower, and the ability to provide security to images that cannot bereconstituted to determine identity. Thus, it is preferable inaccordance with an embodiment of the present disclosure to store enoughdata that future analysis and processing on the feature data is possiblewhile providing sufficient security to protect the identity of anyindividual included in an image.

As is therefore shown in FIG. 2, similar to graph 100 in FIG. 1, on the“x” axis 310 is the storage of information, with the far right equatingwith storing complete images (i.e. all of the data). The “y” axis 320describes computational cost of processing data. In a situation whereall images were stored, but they cannot be stored indefinitely (forexample), ideal data processing would take place at point 330, where theanswers to all possible questions were pre-calculated, before deletingof the data. Not only is this quite computationally expensive, it isalso not possible to know each and every question ahead of time. Area340 on the chart identifies a selection of information where the storedinformation (features) result in de-identified, secure information, butwhere in the future, this information may still be processed in order toprovide additional insight as requested. Any data storage configuration350 residing in area 340 will meet these criteria. Therefore, inaccordance with a preferred embodiment of the present disclosure, theproper number of features to be stored is determined, data is storedusing these features, and then in the future, additional processing maybe performed on the stored data. As noted, the location of configuration350 may be determined by a user or in accordance with an automatedprocess or other testing.

Such testing or processing may provide one or more guidelines as to thenumber of features to be extracted from one or more images and to bestored in order to support any future processing. The images to beprocessed may be further classified by complexity or the like todetermine the number and type of features to be extracted (i.e. morecomplex images may result in more features to be extracted, representingthe richness of information available, while less complex images mayonly warrant extraction of a fewer number of features). Once suchclassifications are determined, they may similarly be used to classifyand process future images in a more streamlined process. Thus, forimages of certain complexity, features determined to be helpful in theprocessing of prior images including a similar complexity may again beused. Alternatively, testing may be performed during live processing oneach image, or on a subset of images, to adjust on the fly the numberand type of features to be extracted.

As noted with respect to FIG. 2, in addition to addressing the abilityto currently process information based upon available processing power,and the ability to provide security to images that cannot bereconstituted to determine identity, the subject matter of the presentdisclosure relates to systems in which the feature or other data storedin accordance with the selection of features to be stored for each imageallows for the future processing of image data without maintaining theactual complete image. Thus, as described above, in a situation wherethe feature data is stored and the full image is deleted, it ispreferable in accordance with the present disclosure to store enoughdata that future analysis and processing on the feature data ispossible. In such a manner, image data may be stored in a secure,de-identified manner, but also allowing for the ability to reprocess thedata in the future should new analysis methods be developed, or shoulddifferent questions be asked of the data.

In an alternative embodiment of the present disclosure, rather thanpredetermining a number of features to be extracted from an image, ormaking such determination in an automated fashion during processing ofan image, a mechanism for determining a number of features/genericlayers to be extracted may be provided to a user, such as a slider,dial, or number selector. Thus, the user may be provided with a visualmechanism for selecting the relationship between security and level ofinformation. For each image or each batch of images, based upon thesubject matter thereof, the user is able to select the number offeatures stored, and thus the tradeoff between security and dataavailability. The system may preferably determine a most useful set ofstored features, or a level of granularity may be provided to users toidentify any desired bias in the selection of such features. Forexample, if the user is likely to want to answer questions in the futurefocusing on a particular area, features may be chosen that are morelikely to help in answering such questions.

In accordance with a preferred embodiment of the present disclosure,various processing may be performed on the one or more images toidentify one or more features of the subject of the image. In a morepreferred embodiment, the subject may be the face of a person, and thefeatures may comprise one or more keypoints or landmarks, and mayfurther comprise one or more known predetermined and mapped keypoints orlandmark points, or time-domain features such as significant activities.Alternatively, such features may be determined through analysisemploying an unsupervised learning process, a supervised learningprocess, or other teaching or learning process.

It should be noted that the ability to process these features, includingone or more keypoints or landmark points may be limited by the availableprocessing power of the associated computer processor. Thus, forexample, while a processor associated with a mobile device may be ableto extract “x” features or points, a dedicated processor may be able toextract 10×, 100× or more features or points. Therefore, it is importantthat any processing system be able to address these varying number ofpoints.

Referring next to FIG. 3, an exemplary embodiment of the presentdisclosure is shown. In FIG. 3, an initial image of an individual 510may be captured by a camera or other image capture device, whether as astand-alone image, or as an image as part of a sequence of images, suchas in a video. As is evident from FIG. 3, when reviewing image 510, itis possible to determine the identity of the subject. Image 520 is apost-processing image derived from image 510, and includes a high numberof extracted features, or in this case keypoints or landmark points. Byno longer storing the complete image data, but rather storing a set ofthese points, data storage requirements can be greatly reduced.Additionally, such pre-processing may allow for faster computations whenfurther processing the data. As can be seen from image 520, however, itis still possible to determine the identity of the person originallyshown in image 510.

Moving next to image 530, another post-processing image derived fromimage 510 is shown. The number of points stored associated with theimage has been reduced from the number shown in image 520. As can beseen, while the major features are still visible, some of theidentifying detail has been removed. It may still, however, be possibleto identify the individual in image 510 by looking at image 530.Additionally, with further computer processing, it may be possible toregenerate something approximating image 510 from image 530.

Fewer points are stored and used to generate image 540, and indeed, onecan see that the ability to determine the identity of the subject in theimage is reduced. Image 550 relies on even fewer points, and may beconsidered de-identified, as there is not enough stored information togo back to the image 510.

It is important to note that the points stored in image 550 arepreferably a subset of the points stored in image 540, which are asubset of points in image 530, and then 520. Alternatively, and moregenerally, a relationship of some sort is preferably defined between thekeypoints/features in the different images. By determining the mostcritical points for determining action, recognition or the like, thesepoints can be prioritized in image 550, but also stored in images 540,530 and 520 to allow for consistency of processing.

As noted, in a situation where different numbers of features or pointsare extracted, the fewer points are preferably a subset of the extractedlarger number of points. Thus, when processing is to proceed, it ispossible to combine images having a different number of extractedpoints. Processing may be variable based upon the number of points, ormay be based upon the minimum number of points in one of the images tobe processed, these same minimum number of points being processed fromeach stored set of points representative of an image, and irrespectiveof whether additional points may be available for processing.

As described with reference to FIG. 3 (and also considering FIG. 2), inthe event that a very large number of points are extracted from an image(i.e. image 520), it is possible to maintain a high quantity of relevantinformation. While ideal for potential future processing, this situationalso results in maintenance of the identity of the subject of the image.On the other hand, if only a few points are extracted (i.e. image 550)it is less likely that the identity of the person can be reconstructed.In accordance with a preferred embodiment of the present disclosure, apredetermined number of features may be preferably identified that allowfor future processing of data, but result in a lossy dataset so that theidentity of a subject in an image cannot be determined, and theinformation necessary to do so is not recoverable. By properly selectingthe correct number and universe of points to be extracted, a balancebetween information (and thus future ability to reprocess the visualinformation), and security (having lost enough data to not allowidentity to be revealed) may be achieved. While FIG. 3 depicts selectingfeatures and points from a face, extracting such features from a hand,other body part, or any other portion of a scene of an image may beemployed.

Thus, the selection of the number of types of features, or points to beselected may be performed in a simple manner (i.e. limiting to aparticular predefined subset of points to be extracted), or may beperformed in a more complex iterative manner, such as by evaluating theability to reconstitute information given a set of features, and thenadjusting whether to extract a greater or fewer number of points, oreven whether to extract different points, regardless of the number ofpoints. Additionally, various external information may be used in orderto guide in point selection. For example, if a particular population isincluded, this may result in particular keypoints and features beingselected. For instance, if a patient population has an issue with facialtwitching, then in accordance with a preferred embodiment of the presentdisclosure, features selected may be one or more that allow for anin-depth analysis of any twitching on the face. Other desired featuresmay also be employed in order to extract particular information for usein a particular analysis.

In addition to selecting one or more keypoints or other features in thevisible spectrum, non-visible light, or other areas on theelectromagnetic spectrum may preferably be employed. The use of suchnon-visible and other electromagnetic radiation allows for theextraction of additional information from an image, and for example theface of a person in an image. It is also contemplated in accordance withone or more embodiments of the present disclosure that differentfeatures or other keypoints may be selected in accordance with thedifferent forms of electromagnetic radiation. Various information andanalysis may be performed in accordance with each of these multipleforms of radiation, the results thereof interacting to provide yetadditional information in determining one or more features. Variouscomputational photography techniques may be employed, utilizing socalled “edge of camera spectrum” to formulate a greater and much higherresolution understanding of, for example, blood flow and blood pressure.Such computational photography techniques can thus be utilized to buildupon derivative data sets. Indeed, any applicable features may beemployed, either alone or in combination, to extract valuable data.

In accordance with one or more embodiments of the present disclosure, itis anticipated that the use of one or more sections of theelectromagnetic spectrum will allow for an in-depth analysis of facialor other visible user features. For example, rather than simply notingexternal facial features, use of the techniques and systems disclosedherein allow for the determination of the location of various bloodvessels under the skin of a user in the field of view of a camera. Overtime, differences determined in the various images provide informationabout the performance of the user, and may further indicate changes indisease, physical ability, or the like. Such changes, for example, maybe more visible under near-infrared light, or other wavelength ofenergy, thus resulting in additional information being extracted basedupon the use of multiple types of light, energy, or other dataextraction mechanisms.

Thus, by overlaying multiple layers of information, irrespective of theamount of information provided in each of those layers, a more concretepicture of the status of an individual may be provided. Informationcollected from lower power devices may include lower resolutioninformation, such as images with fewer pixels, or may include differentmechanisms for data collection. As more information is acquired, eitherthrough different collection mechanisms, or as the power of collectionimproves over time, correlations between collected data and negativeoutcomes (or positive outcomes) in one or more medical therapeutic areasmay be provided. The system may therefore be employed as a diagnostictool for predicting disease or other diagnosis based upon processing ofvisual images of a user. By allowing for the use of such varied layers,the system is robust to methods of collection, timing of collection, andstorage formats of the information, thus “future proofing” the system torely on older data (perhaps at lower resolutions), and newer collecteddata, having a higher resolution, or a newer or new data collectiontechnique.

As is shown in FIG. 4, any number of sensors 410 (in addition to visualsensors) may be provided to collect corresponding pieces of information.As is shown in the exemplary embodiment shown in FIG. 4, one or more ofsuch sensors may, without limitation and by way of example only, measureone or more layers/features, such as shape, light reflection(photometric), texture, other generic sensor, metadata, motion, auditoryinformation, structure, etc. The use of these inputs, combined indifferent manners, results in the ability to determine one or moreconditions/quantities 420, including, without limitation and by way ofexample only, heartrate, temperature, blood pressure, body mass,tremors, action units, gaze direction, skin color changes, etc. Theseconditions/quantities 420 may in turn be combined in order to diagnoseone or more conditions, including, without limitation and by way ofexample only, Parkinson's progression, depression severity, hepatitisprogression, diabetes progression, mood, inflammation, pain level, etc.

Processing included in FIG. 4 may be utilized to explore potentialdiagnosis 430 upon measurement of information from sensors 410, or mayseparately be utilized to identify an expected potential diagnosis 430,and in turn determine which conditions/quantities must be measured toconfirm the diagnosis, and in turn which sensors 410 must be used tomeasure appropriate input information to confirm the conditionsquantities. Thus, the system may be employed to measure multitude ofquantities, and in turn, determine a diagnosis (moving from left toright in FIG. 4), or may be employed to confirm a diagnosis, thusrequiring measurement of only the critical conditions and information(moving from right to left in FIG. 4). Therefore, in accordance with theembodiment of the present disclosure noted in FIG. 4, one or more sensorinformation may be selected to be stored in order to allow for theanswering of future questions.

Therefore, in accordance with the embodiment of the present disclosurenoted in FIG. 4, one or more sensor information may be selected to bestored (See FIG. 2) in order to allow for the answering of futurequestions while avoiding the storage of any identifying information.Through the proper selection of conditions/quantities to be measured,this information may be stored while avoiding storing all visualinformation of an image, or from one or more other sensors.

In accordance with an embodiment of the present disclosure, a user maybe monitored performing one or more activities, such activities havingbeen determined to be indicative of existence or progression of disease.Thus, by monitoring a user in accordance with a visual record, or othernon-visual record, one or more characteristics may be monitored todetermine whether a particular user may be diagnosed with a disease, orwhether a user with a particular disease has one or more characteristicsthat are progressing over time, indicative of a progression of disease.

Such monitoring may take place in an active or passive monitoringsituation. In an active monitoring situation, as is shown in FIG. 17,the user may be asked to perform a particular set of actions, and ispreferably led through these actions on a mobile or other local deviceat step 1710. A display associated with the local device displays one ormore instructions to the user, and then captures (visual, audio, etc.)information related to the performance of the actions by the user atstep 1720. Thus, if the user is to perform an eye movement test, theuser may be instructed to watch a marker on a display, for example. Sucha marker may then be displayed on the display, and moved in apredetermined sequence around the display. This feature thus measuresability to maintain focus on a single item moving through a field ofview. Monitoring or eye movement (“gaze tracking”) may be employed atstep 1730 to determine disease, or if performed over time, may beemployed to determine progression of disease as is shown at step 1740.

In an alternative embodiment, a user may be asked to focus on aparticular marker on the display at step 1710, and a second marker maybe provided on the display, and the ability for the user to continue tofocus on the initial marker may be measured at step 1730. This featurethus measures ability to maintain focus, and how easily the user isdistracted. Again, monitoring over time allows for a determination ofprogression of disease at step 1740.

In a still further embodiment, the user may be monitored withoutproviding a particular stimulus on the display. For example, the abilityfor a user to consistently read text on a screen, or otherwise focus onitems in the world around them may further be monitored in a passivemanner.

In yet another embodiment, an augmented reality or virtual reality scenemay be presented to a user, and a response to the presented informationmay be monitored. Such an augmented or virtual reality schema may beemployed with either active or passive monitoring situation, and maypreferably be employed when encouraging a user to follow one or moresteps associated with monitoring and encouraging proper medicationadministration. (see one or more of U.S. Pat. Nos. 8,731,856, 8,731,961,8,666,781, 9,454,645, and 9,183,601 patents previously incorporatedherein by reference). The augmented reality system may similarly beemployed to provide one or more instructions or other guidance as notedin these applications incorporated by reference, and may further, afteranalysis of collected visual or other data, be used to provideadditional instructions to the user to properly perform one or moredesired sequences, such as properly administering medication, inresponse to determination of an error. The system may be applicable tooral, inhalable, injectable, or any other method of medicationadministrations, or other actions associated with health or healthstatus.

Movement may also be tracked, and similarly may be tracked both activelyand passively. Active monitoring may ask the user to perform apredetermined sequence of steps, and confirm proper performance, andchanges in ability of performance over time. Similarly, passivemonitoring may track gait changes over time, ability to stay seatedwithout fidgeting, etc. The monitoring of either of these movementissues may be employed to diagnose or monitor progression of disease.

Additionally, making reference to U.S. Pat. No. 13/189,518, previouslyincorporated herein by reference, output from any of the schema forpresenting and monitoring a user may be input into a system fordetermining current and potential future medication adherence status ofa user. Therefore, any of the features of the present disclosure may beemployed as inputs into the system as described in the '518 application,allowing for a more robust set of inputs to drive determinations ofadherence and health status, and to drive potential interventions,monitoring and the like of any particular user.

The following additional features, parameters or characteristics may beactively tracked in accordance with one or more embodiments of thepresent disclosure: observe reactions while performing task; Trackpoint; Look at pictures of emotional faces; Coordination test; Balancetest; Reaction speed test; Reflex/reflex suppression; Memory; Keepingappointments (engage with phone at designated time); etc.

The following additional features, parameters or characteristics may bepassively tracked in accordance with one or more embodiments of thepresent disclosure: observe reactions while engaged in other activities(Gaze; Action units; Head motion/pose; Pupil dilation; etc.)

In a further embodiment of the present disclosure, when administering orperforming a therapeutic or recovery exercise, the following additionalfeatures, parameters or characteristics may be actively tracked inaccordance with one or more embodiments of the present disclosure:Tracking motion; Concentration; Memory; ability to focus one's eyes(e.g. after eye surgery to monitor healing timeline). Remote therapy mayalso be incorporated when administering or performing a remotetherapeutic or recovery exercise, or during passive monitoring ofpatients remotely: administering therapeutic/recovery exercises withremote guidance: Track markers while conversing; Symptom tracking;Tremors; Involuntary motions; Jaundice; Temperature; Heart rate; BMI.Recovery tracking may also review one or more of Recovery tracking;Skin/cosmetic surgery; Eye surgery (pupil size etc.), Hair implant, etc.Of course, other characteristics may also be monitored to the extentthat these characteristics are determined to be indicative of diagnosisor progression of disease.

Furthermore, to the extent any such relationship between a measurecharacteristic and disease has not yet been defined, in accordance withan alternative embodiment of the present disclosure, collected data maybe processed to determine any such relationships to allow for use in thefuture. Different demographic groups may be employed to determinecharacteristics in these particular demographic groups, and thus allowfor targeted diagnosis and monitoring of disease. The use of supervisedor unsupervised learning techniques may be employed to analyze such datato determine any applicable relationships.

Referring next to FIG. 5, an additional view of the structure presentedin FIG. 4 is shown. As is shown in FIG. 5 in addition to FIG. 4, sensors410 may take on any number of modalities 450 or modules 455. Modalities450 collect identifiable information while modules collectnon-identifiable information. A combination of this collectedinformation preferably triggers a confirmation of one or more changes inthe noted one or more physical attributes 460, ultimately resulting inan output of a potential diagnosis 465. Analysis of one or more ofmodules 455 may also provide one or more outputs to a user-facingoutput, such as a dashboard or mobile app, related to analysis of themodules 455, thus predicting one or more of, for example withoutlimitation, interaction mood, suspicious behavior, instructions (thatmay be required to assist the user), or screen illumination (which maybe employed in the event of a dark room where video cannot otherwise becollected).

By dividing populations by demographics, disease state, or othermechanism, it is also desirable to provide input into the system as toone or more medical or other conditions that may be more likely or moredangerous in the identified population. While older individuals may bemore likely to suffer a stroke, for example, younger individuals may bemore prone to overuse injuries, or other disease states. By providingsuch information as an input to the system, a reduced set of diseasestates or other changes in parameters may be examined, thus improvingthe accuracy, repeatability and precision of the inventive system.Desired sensors may be employed in order to watch for a particulardisease state, or other indication of disease of a user.

Any such readings determined in accordance with the inventive system mayalso be correlated with one or more known, validated scales or questionsets. Correlations between responses to the scales and featuredetermination may be provided, resulting in yet additional predictiveanalytics, and less reliance on the actual scales going forward. Thus,in accordance with FIG. 4, one or more combination of sensor input datamay be determined to be useful in identifying a particular condition.Once value ranges for each of the desired sensor inputs are determinedthat correspond, as a whole, to a determination of existence of aparticular diagnosis, this combination of sensor input and data valueranges may be stored for future use.

Once various feature sets have been determined to have a predictivecapability (for predicting, for example, existence of a particulardisease, one or more data masks may be generated to allow for a reviewof incoming future data to similarly determine existence of that sameparticular disease, for example. Such a data mask may comprise one ormore feature values indicative of a state of an individual. Thus, forexample, rapid weight gain, an increase in blood pressure, and a changein skin color may be defined as indicative of the progression of aparticular disease. By providing such a mask, a simple diagnostic toolmay be provided that allows for an immediate and automated determinationof potential medical conditions of an individual, and may furtherindicate a preferred path for addressing these conditions. For example,the individual may be given instruction to rest, to drink fluids, toeat, to call their doctor, to take medication, or to go immediately toan emergency room. The system may further be provided with adjunctsystems for performing some of these functions, such as calling a familymember, dialing 911 and providing GPS information for location of theindividual, and may also provide an initial diagnostic analysis of thehealth of the individual to those responding to the alert, allowingquicker response and triage on site, or when the individual reached thehospital.

By way of example only, understanding the artery, capillary and veinstructure of the body may allow for a determination of changes therein,in order to allow for visual measurement of heartrate, temperature, orother bodily measurements, as will be described below with respect toFIG. 6. Thus, in accordance with a further embodiment of the presentdisclosure, it is possible to focus a visual system on a particularportion of the body of a user to determine visually a heartratetherefrom. As noted above, a system employing visual or otherelectromagnetic radiation may be employed. It may be difficult, howeverto determine minimal changes in the skin of a user when the location ofsuch changes is unsure. Thus, the system will be required to scan acomplete user body to look for slight changes, and then determinewhether these changes are significant to any bodily measurement. Anychanges in the collected data may be used to predict changes in health,disease progression, symptom progression, overall health or the like.

As noted by the inventors of the present disclosure, and as a furtherexample, by understanding the artery, capillary and vein structure ofthe body, determination of changes in any such structure may be moreeasily determined. By knowing where to look for, by way of example acapillary at the temple of a user, it is easier to determine changes inthat capillary as the system can be properly focused, and need not scanthe entire body to determine where to look. In order to perform such atask, the inventors of the present disclosure have determined that it ispossible to overlay a data mask indicative of the locations of, forexample, arteries and veins on a typical face, on an image of anindividual. As is shown in FIG. 6, a known face blood vessel atlas 610may be determined from one or more known data sources. Such an atlas mayalternatively be generated in accordance with one or more scans ofactual bodies, or any other composite image from which a blood vesselstructure can be determined. Overlaying face blood vessel atlas 610 on aface image 620 results in the composite image 630. Referring to suchcomposite image 630, the system may be guided where to look for aparticular blood vessel, and any changes thereto. In an additionalembodiment of the present disclosure, a plurality of masks may bedetermined as a subset of the complete mask, based upon desiredlocations for a particular disease state, or the like. Thus, preferablysuch mask may be determined after processing data for multipleindividuals, and may be categorized by demographics, disease state orthe like. The mask may be further customized to a particular individual,and then monitored over time to determine changes in the individual,related to relevant variables, over time. Such changes may be used topredict changes in health, disease progression, symptom progression,overall health or the like.

In accordance with a preferred embodiment of the present disclosure, oneor more particular facial or other features may be captured from imagesin question. As is shown in FIG. 7, feature extraction may take place onfeatures captured in visual layers, additionally features capturedhaving a temporal dimension, or using other layers. FIG. 7 describes aset of potential features, or other image elements that may be used toextract information from one or more video or still images. As is shownin FIG. 7, a number of different layers may be analyzed in order todetermine one or more features of an image that may be used to storeextracted information. These layers may comprise, for example, a shapelayer 710, a photometric layer 720, a textual layer 730, a sensor layer740, and a metadata layer 750. The first four of these elements(710-740) include a temporal dimension 760, while the first three(710-730) additionally include a visual layer 770. Additional layers 780may further comprise a motion layer 785, an auditory layer 790, and astructure layer 795. For each such layer, various features may beinvestigated as to whether they are relevant to a particular set ofimages. For example, if an image has varied lighting conditions, thenthe photometric layer and lighting conditions therein may be used tostore information about an image. If, however, lighting is constant,then this layer and feature will likely store very little informationabout the image, and may not be used.

The layers of FIG. 7 may be used in a number of different ways. First,one or more of the particular layers may be chosen based upon thedesired question to be answered, related to the task at hand. Forexample, if one desires to identify Jaundice, or other skindiscolorations, then the visual color layer may be an important one thatshould be analyzed, while shape features are likely less important.Identification of tremors, on the other hand, may rely on shapeinformation much more heavily than color. Similarly, the determinationof body mass index may rely on shape features, and changes thereto overtime. The above-noted various forms of electromagnetic radiation maysimilarly be employed to allow for additional of these layers to bedetermined. Thus, for example, shape or color may appear differentlyemploying these different forms of electromagnetic radiation.Additionally, reflectivity or other characteristics of the human bodymay be measured in response to these different applications ofelectromagnetic radiation, allowing for additional and variedinformation to be collected. Audio pulses or the like may be employed,to determine water and/or fat content, as well as any other determinablefeature. Additionally, one can extract features from any number of thenoted or other layers, and make a determination of whether thisinformation is valuable. For example, extracting color data only to findthat the screen is completely black will not result in much information(but may indicate that the system has improperly recorded information).Noting that lighting conditions are consistent is also relevantinformation, and may or may not be important depending on the task athand. Therefore, in accordance with various embodiments of the presentdisclosure, layer data may be extracted, analyzed, and determinedwhether helpful in a particular situation. Additionally, if one islooking for a particular type of information, the desired layers foranalysis may be pre-identified. Any of this information may be reviewedover time to determine, for example, weight gain or loss, blood pressureor pulse change, dehydration, frostbite, hypothermia, or any othermeasurable quantity. In addition, one or more of the visual layers maytake advantage of high speed photography, allowing for the collection ofinformation up to 400 frames per second or higher, for example, to allowfurther analysis of very subtle changes in skin movement, color, orother micro gestures that may be of interest.

Therefore, in accordance with various embodiments of the presentdisclosure, layer data may be extracted, analyzed, and determinedwhether helpful in a particular situation. Additionally, if one islooking for a particular type of information, the desired layers foranalysis may be pre-identified. Any of this information may be reviewedover time to determine, for example, weight gain or loss, blood pressureor pulse change, dehydration, frostbite, hypothermia, or any othermeasurable quantity. In addition, one or more of the visual layers maytake advantage of high speed photography, allowing for the collection ofinformation up to 400 frames per second or higher, for example, do allowfurther analysis of very subtle changes in skin movement, color, orother micro gestures that may be of interest. Images may be consideredin chronological order, or may be considered simultaneously, relying oncommonality of structure rather than time. Such processing preferablygroups incoming images along one or more dimensions other thanchronological. In such a manner, images may be grouped in a mannerproviding the most information for determining existence of a particularcharacteristic. Such characteristic may be indicative of existence orprogression of disease, or may be relevant to any other measureablequantity or quality of disease or other characteristic. By groupingincoming images along such non-chronological dimensions, as more imagesare collected, the number of images in each grouping may be increased,thus improving the precision of any processes dependent upon thesegroupings. Any of the layers shown in FIG. 7, or any other applicablelayers may be included as the one or more dimensions, and allow forprocessing of images in a manner most helpful for determination of imagestatus, and in accordance with a preferred embodiment of the presentdisclosure, helpful in determining state and progression of disease.Thus, captured layers and dimensions may be related to, for example,skin color, eye movement, body temperature, heart rate, or the like.

Therefore, further in accordance with a preferred embodiment of thepresent disclosure, the one or more layers (either visual or having atemporal dimension, such as layers working with various non-visualsensors, or even metadata), may be examined to determine whether one ormore features thereof is relevant to a particular set of images. Oncedetermined to store important information, these features within asingle layer, or across multiple layers may be grouped and selected as asubset of keypoints or other information, in a fashion as noted above,in order to select a subset of the possible information, to allow forstorage and processing of the data in the future, but preferably in alossy manner in order to protect the identity, and reduce the ability tore-identify an image. Of course, this process may also be employed evenif an image is to be stored in a lossless manner, or in a lossy mannerthat still allows for re-identification of information stored therein.In addition to a formal visual layer, it is also to further processadditional types of electromagnetic radiation, such as ultraviolet,infrared, etc. In such a manner, it is possible to extract features thatmay not be visible under electromagnetic radiation in the visual portionof the spectrum, but may be more prominent once other areas of thespectrum are used.

Referring next to FIG. 8, a sequence of particular features that may becollected when imaging visual and other information related todetermination of whether a particular user has properly administeredmedication, as noted in the above patents incorporated herein byreference. As is shown in a preferred embodiment of FIG. 8, a number ofsteps indicative of proper medication administration are shown,including proper positioning of a capture device 210, facial recognition220, object recognition 230, pill in mouth recognition 240, empty mouthrecognition 250 and under tongue empty mouth check 260. As is furthershown, in FIG. 8, keypoint tracking 271 may be employed across all stepsstarting with facial recognition. Tracking keypoints associated withmedication in hand 272 preferably takes place only during objectrecognition 230, while accelerometer data 273 may be used at all timesto be sure the capture device is properly positioned. Finally, facedescriptors 274 may be employed at any time the face of a user is withinthe field of view of the capture device.

Such processing may be applied, for example to images of a useradministering a pill or film based oral medications, or injectable,inhaler-based, other non-pill based medication, or any other form ofpatient administration task that may be performed, may be utilized inaccordance with one or more of the present disclosures noted in theabove-referenced applications. Therefore, in accordance with anembodiment of the present disclosure, a method and apparatus may beprovided for analyzing captured patient image data.

The system may process information at a remote system housing thedatabase of collected information. New images acquired by a camera in alocal mobile device may be transmitted to the remote location, one ormore of the above-noted processing techniques, including extraction ofkeypoint/feature data may be applied, and then the results of suchanalysis may be provided as feedback to a user in any number ofscenarios. Such keypoint/feature processing data may be used todetermine one or more features of medication adherence, facialrecognition, facial images, or the like based upon differing levels ofstored information and keypoint/feature data. Alternatively, as notedabove, processing may be performed in advance of new image acquisitionto identify a desired number of keypoints/features to be extracted, andthen a system for identifying characteristics of a future image may beprovided to the local device to further process acquired images. Such asystem may be reduced in processing requirements to allow forappropriate processing on the local mobile device. By providing eitherof these systems in accordance with a medication monitoring process,when monitoring medication administration, various features of theadministration process may be extracted.

A mechanism for determining a number of features to be extracted may beprovided to a user, such as a slider, dial, or number selector. Thus,the user may be provided with a visual mechanism for selecting therelationship between security and level of information. For each imageor each batch of images, based upon the subject matter thereof, the useris able to select the number of features stored, and thus the tradeoffbetween security and data availability.

As noted above, in one embodiment of the present disclosure, in asituation where different numbers of features or points are extracted,the fewer points are preferably a subset of the extracted larger numberof points/features, as described above. Thus, when processing is toproceed, it is possible to combine images having a different number ofextracted points/features. Processing may be variable based upon thenumber of points, or may be based upon the minimum number ofpoints/features in one of the images to be processed, these same minimumnumber of points/features being processed from each stored set of pointsrepresentative of an image.

By way of example, if the selected features are representative of theface of an individual (FIG. 3), images having a small number of pointsextracted (Image 550) may be representative of the eyes, nose and lipsof the subject, as these may provide the most information. Images havinga high number of features or points may include these points, andfurther include points related to forehead, chin, cheeks, etc. (Image520).

In U.S. Pat. No. 9,256,776, and U.S. patent application Ser. No.14/990,389, the contents thereof being incorporated herein by reference,the owners of the present disclosure have described a system that allowsfor the partial blurring of images to de-identify one or more images ina video sequence. In accordance with the present disclosure, as setforth in FIG. 9, this concept may be combined with the techniques andsystems disclosed herein, whereas the unblurred portion 910 of a user'sface, for example, may be de-identified further using the keypointsystem described above with respect to FIG. 3, while a pill in handportion 920, or a mouth portion 930 may remain unblurred. In such amanner, additional security provided in that any portion of the imagethat is not blurred is de-identified, and any portion of the image thatis not keypoint de-identified, and likely of less interest to theviewer, is therefore blurred. This is a useful feature to insure thatany individual walking through the background of the image is notidentified inadvertently.

De-Identification Models

Referring next to FIG. 10, a de-identification model, including a methodfor generating de-identified images which are compatible with one ormore existing modules, is presented in accordance with a preferredembodiment of the present disclosure is shown. FIG. 10 describes theprocess of de-identifying a particular presented image with a trainedde-identification model (details about training are presented in FIG.11).

As is shown in FIG. 10, at step 1010 an image is preferably acquired inaccordance with an image acquisition process employing an image captureapparatus, such as a camera, associated with a mobile device, standalonecamera, or other image acquisition system. At step 1020 an image of aface is preferably extracted from the acquired image. It is desirable tomask the identity of the face. The facial image is thus preferablypassed to a de-identification model at step 1030. The output image atstep 1040 comprises a de-identified image which preferably appears as anatural image of a person whose identity is public (i.e. the identity ofthe facial image has been transferred into an image of someone elsewhose identity need not be protected, or to a composite facial identitythat is not identifiable as any particular individual). The output imageat step 1040 will retain aspects deemed important for the tasks (knownand unknown) that may be performed (e.g., in a preferred embodiment,layers as noted above will be retained to allow for the answer of futurequestions) but private characteristics are preferably replaced withpublic ones. For example, the pose, expression, skin tone and gazedirection of the individual in the private image may be maintained butother private attributes are preferably synthesized in a manner whichwill prevent identification. The resulting synthesized image thereforeincludes a natural appearance, and is compatible with existing modulesfor known (and unknown) tasks which expect a natural face image. Thus,at step 1050, additional processing may be performed on the output imageas if the image were that of a simple acquired facial image.

Referring next to FIG. 11, a method for training the system of FIG. 10to optimize the parameters of the de-identification model is shown. Theoptimization method preferably follows the processes as described inaccordance with a Generative Adversarial Network (Goodfellow, Ian, etal. “Generative adversarial nets.” Advances in neural informationprocessing systems. 2014.), and self-regularization (Shrivastava,Ashish, et al. “Learning from Simulated and Unsupervised Images throughAdversarial Training.” arXiv preprint arXiv:1612.07828 (2016)).

As is further shown in FIG. 11, an image is acquired at step 1105, andat step 1110 a private-identity facial image is extracted therefrom.Next, at step 1115 a de-identification model receives theprivate-identity facial image and synthesizes a de-identified version ofthat image at step 1120. The de-identified result at step 1120 isevaluated with two loss functions so that the parameters of thede-identification model are optimized to minimize the loss on a largedataset of training examples. Thus, the de-identification process shouldnot modify task relevant features, so that a greater percentage offuture questions can be answered (e.g., in a preferred embodiment, agreater number of layers associated with likely potential questions arepreferably retained). A loss function at step 1125 evaluates retentionof relevant features in a way appropriate to the task at hand. Forexample, in order to verify medication ingestion it may be desirable tominimize changes in the mouth area. This loss may be implemented bymeasuring the distance between the private and de-identified image usinga mask 1130 which assigns more weight to differences in the mouth regionthan to differences in other parts of the image.

The de-identified image at step 1125 may further be combined at step1135 with a public identity image 1140 (or itself a synthesized image)retrieved from a database or other storage location 1145, and providedto a discriminative model element 1150. This element receives thecombined image from step 1135 (which may be a real public identity imageor a synthesized de-identified image) and outputs a binary label at step1155 indicating whether the discriminative model element 1150 hasdetermined that the image is a real or a synthesized image. Thede-identification model is trained to generate realistic image whichfool the discriminative model. The discriminative model is trained todistinguish between the real and synthesized images.

The discriminator performance is then evaluated in accordance with acalculation of a binary classification loss at step 1160. The results ofthe discriminator output are compared with the known combination systememployed in step 1135 to modify settings in a feedback loop system, asappropriate, until the system can consistently fool the discriminator.

Referring next to FIG. 12, a joint optimization de-identification modelconstructed in accordance with an embodiment of the present disclosureis described. The system and method as depicted in FIG. 12 describes ajoint optimization method to generate a de-identified representation ofan image which can be used to perform multiple tasks of interest denotedhere task_1 . . . task_n. The diagram describes the process ofde-identifying with a trained de-identification model (details abouttraining are presented in FIG. 13).

As is shown in FIG. 12, an image is acquired at step 1205, and at step1210 a face is extracted therefrom, the face including an identity thatis desired to be masked. The private image is passed to ade-identification model at step 1215. The output from step 1215comprises a de-identified representation at step 1220. The outputde-identified representation can be used to solve different known tasks,from task model 1 shown at step 1225 to task model n at step 1230. Eachof the task models 1 to n preferably rely on one or more layers ofinformation. Thus as noted above, the layers retained in accordance withthe selected de-identification model at step 1215 are preferablyselected in order to maximize the likelihood of answering one or morefuture questions. If the representation is trained to provide goodperformance for a rich and varied set of known tasks it may also begeneralizable and useful to other unknown tasks which have somecorrelation with the known tasks.

Referring next to FIG. 13, a joint optimization method to train ade-identification model presented in accordance with an embodiment ofthe present disclosure is described. This method jointly optimizes anumber of models in order to minimize a binary classification loss and asequence of task specific losses. The binary classification losspromotes the de-identification while the task specific losses promoteretention of good performance for tasks which rely on the de-identifiedrepresentation.

A pair of images (image_1 and image_2) with known identities areretrieved from a dataset of identity annotated facial images 1305 atsteps 1310 and 1315. At step 1320, a flag is set to indicate whether theimages at step 1310 and 1315 are of the same person or not. Next, atstep 1325 a de-identification model is applied to each of the imagesimage_1 and image_2, outputting de-identified representations 1 and 2 atsteps 1330 and 1335 respectively. These two de-identifiedrepresentations 1330, 1335 are fed into a same-not-same model 1340.Same-not-same model 1340 estimates whether representations 1330, 1335are of images of the same person or not, outputting the result at 1345regarding its decision. At binary classification loss step 1350, theoutput at step 1345 is compared with the label generated at step 1320 todetermine whether there is a match (i.e. if the images are determined tobe of the same person in step 1345, are they actually of the same personas noted in step 1320. If there is no same-not-same model that performsbetter than random chance, then it can be determined that thede-identification is perfect.

The same-not-same model and the de-identification model described above“compete” with each other. The de-identification model tries to mask theidentity (making it harder to determine what is going on, while thesame-not-same model tries to determine what is going on between theimages. FIG. 14, depicts a possible approximation for joint optimizationof the two models, and operates in a block-wise optimization of theparameters. The system preferably alternates 1) fixing the parameters ofthe de-identification and training a same-not-same model to distinguishidentities in its outputs and then 2) freezing the same-not-sameparameters and optimizing the de-identification model to “fool” thefixed same-not-same model. At 1405 a set of task annotated face imagesare shown. Each image 1410 is extracted and fed through a proposedde-identification model 1425. One or more task labels 1415, 1420 arealso determined for each image. A de-identified representation of theimage 1430 is extracted from de-identification model 1425, and aparticular one or more task models 1435 are performed. A resultingestimate 1440 for each task model is determined, and at step 1445, eachresult is compared to a corresponding task label 1415, 1420 to determinea minimum number of task losses. Once the task losses have been reducedas much as possible, the system has been properly calibrated.

Thus, models may be trained for known tasks 1435 (task_1 . . . task_n)using supervised data. i.e. data which includes annotation for the tasksof interest. The task models are optimized to minimize losses 1445(loss_1 . . . loss_n) which are appropriate losses for each one of thetasks of interest. A possible strategy to jointly optimize the taskmodels and the de-identification model is block-wise coordinateoptimization. The de-identification model parameters may be fixed, andthe task models trained to minimize the sum or weighted sum of theindividual task losses. Given fixed task models, the parameters of thede-identification model may be adjusted to improve performance on thetask losses.

Therefore, in accordance with various embodiments of the presentdisclosure system and method are provided in which, among other results,images of faces can be de-identified in a manner that retains relevantinformation so that a maximum number of future questions can be properlyanswered.

Use of the system may be employed in a controlled or natural setting,and therefore may improve patient engagement, as the system may be usedat any location. Additionally, the user's environment may be measuredin, for example, a recovery situation, to determine whether user is in amost desirable environment to aid such recovery.

Therefore, in accordance with various embodiments of the presentdisclosure system and method are provided in which, among other results,images of faces can be stored in a manner that retains relevantinformation so that a maximum number of future questions can be properlyanswered.

This specification uses the term “configured” in connection withapparatuses, processors modules, systems and computer programcomponents. For a system of one or more computers to be configured toperform particular operations or actions means that the system hasinstalled on it software, firmware, hardware, or a combination of themthat in operation cause the system to perform the operations or actions.For one or more computer programs to be configured to perform particularoperations or actions means that the one or more programs includeinstructions that, when executed by data processing apparatus, cause theapparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them.

Embodiments of the subject matter described in this specification can beimplemented as one or more computer programs, i.e., one or more modulesof computer program instructions encoded on a tangible non transitoryprogram carrier for execution by, or to control the operation of, dataprocessing apparatus. Alternatively, or in addition, the programinstructions can be encoded on an artificially generated propagatedsignal, e.g., a machine-generated electrical, optical, orelectromagnetic signal, that is generated to encode information fortransmission to suitable receiver apparatus for execution by a dataprocessing apparatus. The computer storage medium can be amachine-readable storage device, a machine-readable storage substrate, arandom or serial access memory device, or a combination of one or moreof them. The computer storage medium is not, however, a propagatedsignal.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.The apparatus can include special purpose logic circuitry, e.g., an FPGA(field programmable gate array) or an ASIC (application specificintegrated circuit). The apparatus can also include, in addition tohardware, code that creates an execution environment for the computerprogram in question, e.g., code that constitutes processor firmware, aprotocol stack, a database management system, an operating system, or acombination of one or more of them.

A computer program (which may also be referred to or described as aprogram, software, a software application, a module, a software module,a script, or code) can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment. A computer program may, butneed not, correspond to a file in a file system. A program can be storedin a portion of a file that holds other programs or data, e.g., one ormore scripts stored in a markup language document, in a single filededicated to the program in question, or in multiple coordinated files,e.g., files that store one or more modules, sub programs, or portions ofcode. A computer program can be deployed to be executed on one computeror on multiple computers that are located at one site or distributedacross multiple sites and interconnected by a communication network.

As used in this specification, an “engine,” or “software engine,” refersto a software implemented input/output system that provides an outputthat is different from the input. An engine can be an encoded block offunctionality, such as a library, a platform, a software development kit(“SDK”), or an object. Each engine can be implemented on any appropriatetype of computing device, e.g., servers, mobile phones, tabletcomputers, notebook computers, music players, e-book readers, laptop ordesktop computers, PDAs, smart phones, or other stationary or portabledevices, that includes one or more processors and computer readablemedia. Additionally, two or more of the engines may be implemented onthe same computing device, or on different computing devices.

The processes and logic flows described in this specification can beperformed by one or more programmable computers executing one or morecomputer programs to perform functions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application specific integrated circuit). For example, the processesand logic flows can be performed by and apparatus can also beimplemented as a graphics processing unit (GPU).

Computers suitable for the execution of a computer program include, byway of example, general or special purpose microprocessors or both, orany other kind of central processing unit. Generally, a centralprocessing unit will receive instructions and data from a read onlymemory or a random access memory or both. The essential elements of acomputer are a central processing unit for performing or executinginstructions and one or more memory devices for storing instructions anddata. Generally, a computer will also include, or be operatively coupledto receive data from or transfer data to, or both, one or more massstorage devices for storing data, e.g., magnetic, magneto optical disks,or optical disks. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, e.g., a mobiletelephone, a personal digital assistant (PDA), a mobile audio or videoplayer, a game console, a Global Positioning System (GPS) receiver, or aportable storage device, e.g., a universal serial bus (USB) flash drive,to name just a few.

Computer readable media suitable for storing computer programinstructions and data include all forms of non-volatile memory, mediaand memory devices, including by way of example semiconductor memorydevices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks,e.g., internal hard disks or removable disks; magneto optical disks; andCD ROM and DVD-ROM disks. The processor and the memory can besupplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back end, middleware, or front end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various system modulesand components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

It should be noted that any of the above-noted techniques and systemsmay be provided in combination or individually. Furthermore, the systemmay be employed in mobile devices, computing devices, cloud basedstorage and processing. Camera images may be acquired by an associatedcamera, or an independent camera situated at a remote location.Processing may be similarly provided locally on a mobile device, or aremotely at a cloud-based location, or other remote location.Additionally, such processing and storage locations may be situated at asimilar location, or at remote locations.

1.-18. (canceled)
 19. A method for monitoring a state of an individual,the method comprising: obtaining a video record of the individualperforming a physical activity; de-identifying the video record, whereinde-identifying the video record comprises identifying a subset of pointsto be extracted from the video record, the subset of points allowing forfuture analysis of the physical activity, and storing the subset ofpoints without storing the video record to inhibit re-identification ofthe individual in the video record; measuring the physical activity asrecorded by the stored subset of points; comparing the measured physicalactivity to an expected physical activity to obtain a comparison result;and diagnosing one or more aspects of disease based on the comparisonresult.
 20. The method of claim 19, comprising storing non-visualinformation associated with the video record and selected from one ormore sensor outputs.
 21. The method of claim 19, comprising: identifyinga subset of sensors of a plurality of sensors applicable to recognize aparticular disease; recording a second physical activity by the subsetof sensors; and analyzing the recorded second physical activity toconfirm a presence or absence of the particular disease.
 22. The methodof claim 19, comprising: recording a second physical activity by aplurality of sensors; and analyzing the recorded second physicalactivity to determine a presence or absence of one or more possiblerecognized diseases.
 23. The method of claim 19, wherein the subset ofpoints comprises facial keypoints.
 24. The method of claim 19, whereinthe subset of points corresponds to one or more portions of a face ofthe individual.
 25. The method of claim 19, comprising displaying, to auser, an interface by which the user is able to modify a number ofpoints to be included in the subset of points.
 26. The method of claim19, comprising: blurring a portion of the video record; and storing theblurred portion in association with the stored subset of points.
 27. Asystem for monitoring a state of an individual, comprising: a videocapture device; one or more processors; and a memory storing one or morenon-transitory, computer-readable instructions that, when executed bythe one or more processors, cause the one or more processors to performoperations comprising: obtaining, from the video capture device, a videorecord of the individual performing a physical activity; de-identifyingthe video record, wherein de-identifying the video record comprisesidentifying a subset of points to be extracted from the video record,the subset of points allowing for future analysis of the physicalactivity, and storing the subset of points without storing the videorecord to inhibit re-identification of the individual in the videorecord; measuring the physical activity as recorded by the stored subsetof points; comparing the measured physical activity to an expectedphysical activity to obtain a comparison result; and diagnosing one ormore aspects of disease based on the comparison result.
 28. The systemof claim 27, wherein the operations comprise storing non-visualinformation associated with the video record and selected from one ormore sensor outputs.
 29. The system of claim 27, comprising a pluralityof sensors, wherein the operations comprise: identifying a subset ofsensors of the plurality of sensors applicable to recognize a particulardisease; recording a second physical activity by the subset of sensors;and analyzing the recorded second physical activity to confirm apresence or absence of the particular disease.
 30. The system of claim27, comprising a plurality of sensor, wherein the operations comprise:recording a second physical activity by the plurality of sensors; andanalyzing the recorded second physical activity to determine a presenceor absence of one or more possible recognized diseases.
 31. The systemof claim 27, wherein the subset of points comprises facial keypoints.32. The system of claim 27, wherein the subset of points corresponds toone or more portions of a face of the individual.
 33. The system ofclaim 27, comprising a display, wherein the operations comprisedisplaying, to a user, by the display, an interface by which the user isable to modify a number of points to be included in the subset ofpoints.
 34. The system of claim 27, wherein the operations comprise:blurring a portion of the video record; and storing the blurred portionin association with the stored subset of points.